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ABSTRACT 

A survey was conducted of the literatiire pertaining 
to higher education program review. Many institutions of higher 
education have come under intense scrutiny and criticise, and have 
been required to respond to public pressure for accountability. 
Program review is one component of the assessment of institutional 
effectiveness. The following seven themes were identified as 
providing insight into contemporary program review processes: (1) 
purposes of program review; (2) different approaches to program 
review; (3) program review models; (4) common elements in successful 
evaluations; (5) criteria used in program review; (6) processes used 
in program review; and (7) utilization of results. Minor issues 
related to program review that were cited in the literature include 
common myths, political realities, the state's role in the process, 
and the links to accreditation. (Contains 53 references.) (SLD) 
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INTRODUCTION 



Within the last decade, the role of academic program 
review has emerged as one of the most prominent issues in 
American higher education. Many faculty, administrators, 
governing boards, and state higher education agencies have 
become deeply, if not sometimes passionately, involved in 
program review (Conrad and Wilson, 1985:1), 

American higher education has been publicly criticized in 
recent years. Many institutions have come under intense scrutiny 
and criticism. They have been required to respond to public 
pressures for accountability by improving their programs to meet 
the educational needs of their constituents (Angel, 1990), The 
national accountability movement is not new; however, "only 
recently have colleges taken the initiative to assist in defining 
the criteria by \/hich their effectiveness is assessed" (Rooney 
and Tucker, 1990:vii). 

The term "institutional ef feet iver ess" is considered an 
integrating concept for several interrelated issues, such as: 
quality control, accountability, assessment, and other measures 
of institutional vitality, "If institutional effectiveness is to 
be achieved, there must be some sense of what it is and what it 
would look like" (Kreider, 1990:iv). The assessment of an 
institution's effectiveness requires a systematic evaluation of 
its performance in relationship to its stated missions. Program 
review is a component of the assessment of institutional 
effectiveness . 

The ability to draw on a large number of soundly designed 
and executed studies adds great strength to the knowledge 
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base when findings are consistent across different studies 
conducted by different analysts using different methods. No 
single study, no matter how good, can have this kind of 
power (U.S. General Accounting Office, 1983:1). 

The survey of the literature pertaining to higher education 

program review and evaluation processes contained seven themes: 

purposes of program review; different approaches to program 

review; program review models; common elements in successful 

evaluations; criteria used in program review; processes used in 

program review; and utilization of results. Although program 

review is not considered to be an exact science, these seven 

themes provide insight into contemporary program review 

processes . 

Purposes of Program Review 

Many purposes for implementing a program review process 
exist. One of these purposes is to assess the quality of the 
academic offerings of the institution (George, 1982; Caruso, 
1985; Conrad and Wilson, 1985). Program review can "provide 
valuable insights into how programs are operating, the extent to 
which they are serving their intended beneficiaries, and their 
strengths and weaknesses" (Herman, Morris, and Fitz-Gibbon, 
1987:11). George (1982:50-51) indicated that the assessment of 
program quality is a three-step process: (1) establish a set of 
goals for the program; (2) identify the resources, processes, and 
input variables germane to the established goals; and (3) 



determine how to measure each of the relevant variables in the 
program . 

Talmadge noted that a major purpose of evaluation, closely 

related to the assessment of program quality, was "to render 

judgement on the worth of the program" (1983:18) • Guba and 

Lincoln (1981:45-46) indicated that worth, an intrinsic property 

of program review, is determined by comparing program outcomes to 

externally generated standards, thus legitimizing the program. 

House commented on this legitimization aspect of worth: 

The legitimization function of an evaluation is greatly 
neglected. The most important fact about a university 
evaluation plan may be that the university has one and that 
it is going about its business in a demonstrably deliberate 
and responsible manner. One must publicly justify what one 
is doing (House, 1982:6) . 

A well-designed program review process can improve 
management decisions by assisting decision-makers in the 
determination of whether or not the program is providing relevant 
educational experiences to its consumers (Talmadge, 1983; Larson, 
1985; Wenrich, 1988; Conrad and Wilson, 1985). The effective use 
of a program review process can assist managers in (1) setting 
priorities, (2) providing guidelines for the allocation of 
resources, and (3) facilitating program improvement (Herman, 
Morris, and Fitz-Gibbon, 1987:11). A study commissioned by the 
American Council on Education reported that 91 percent of college 
administrators believe that the results of a program review 
process should be linked to instructional improvements (El- 
Khawas , 1986) . 



Another purpose of program review, closely related to the 
assessment of program quality, is program improvement (Borchers, 
1986; Finley, 1988; Barak and Breier, 1990; Caruso, 1985; Conrad 
and Wilson, 1985). The Commission on Colleges of the Southern 
Association of Colleges and Schools (SACS), in its Criteria for 
Accred i tat i on states the primary concern of accreditation is "the 
improvement of educational quality throughout the region" 
(1989:7). SACS places special .emphasis on the relationship that 
exists between the assessment of institutional effectiveness and 
improvement of programs. Effective institutions use assessment 
information to improve their academic offerings ( Resource Manual 
on Institutional Effectiveness , 1989). Accrediting bodies can 
stimulate program improvement via the requirement that the 
college conducts "periodic self-evaluation to identify what it 
does well, to determine where improvement is needed, and 
develop plans to addres-s those improvements" (Ellison and Smith, 
1987: 1) . 

Several authors listed the elimination of inefficient 
programs as a purpose of program review (Borchers, 1986; Barak 
and Breier, 1990; Larson, 1985; Caruso, 1985). For example, 
thirty-nine Illinois community college programs were discontinued 
in 1987 as a result of program review processes (Illinois 
Community College Board, 1987). A critic of this purpose of 
program review stated "the over-riding purpose of gathering data 
is to provide a basis for improving inst rue t ion, rather than 
keeping score or allocating blame" (Turnbull, 1985:25). 



Finally, a purpose of program review is to demonstrate 
accountability to the program's significant publics. An 
institution that systematically evaluates it's programs can 
assure local and state legislators, federal officials, 
accrediting bodies, and students that it is proceeding on sound 
managerial principles (Borchers , 1986; Talraadge, 1983; Barak and 
Breier, 1990; Conrad and Wilson, 1985). "Institutions have an 
obligation to all constituents to evaluate effectiveness and to 
use the results in a broad-based, continuous planning and 
evaluation process" ( Criteria for Accreditation . 1989-90:13). 

Different Approaches to Program Review 

Conrad and Wilson (1985:17-18) listed three general 
approaches to evaluation that emphasize the role of those 
reviewing the program: internal, external, and multiple. 
Internal review, also known as self-review, is built on the 
assumption that the faculty of a prograu can best assess the 
strengths and limitations of their program. The external 
approach to program review allows reviewers to be chosen either 
from within the college (external to the program being evaluated) 
or from outside the college. The multiple review approach uses a 
combination of internal and external reviews conducted on a 
separate basis. 

House (1982:6-12) emphasized four general approaches to 
evaluation: systems analysis, behavioral objectives, profes- 
sional review, and case study. In the systems analysis 
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approach, a few appropriate outcome measures for the program are 

defined, with difference^ in programs being related to variations 

in the outcomes measures. To illustrate this approach, House 

provided the following example: 

One might reason that a purpose of programs in higher 
education is to maintain some minimum level of efficiency. 
A state board might collect and compare costs per student 
across various institutions and programs. Costs that are 
unusually high might be singled out for closer examination 
(1982:6) . 

The purpose of this approach is to determine if the expected 
program effects are being achieved. The major disadvantage in 
using this approach is that educational programs rarely "lend 
themselves to being measured by a few simple quantitative 
outcomes" (1982:8) . 

In the behavioral objectives approach to evaluation, a 
statement of objectives for the program are specified. 
Evaluation consists of determining whether or not the stated 
objectives have been achieved, House provided the example of the 
program objective to "graduate a particular percentage of the 
students enrolled, subject to certain constraints" (1982:8). 
This approach allows the program faculty to specify unique 
objectives on which the program will be evaluated. The major 
disadvantage in using this approach is that the objectives are 
often arbitrarily chosen, based on only a few of a program's 
ob j ec t i ves . 

The professional review approach utilizes visits by 
accreditation teams affiliated with an academic field of study. 
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The case study approach involves the writing of a narrative of 
the program by an evaluator. The narrative is usually the result 
of interviews with faculty, students, and administrators 
associated with the program. In the professional review 
approach, the experience of the external evaluators is relied 
upon. In the case study approach, the experience of the 
program's participants is relied upon. 

Barak and Breier (1990:28-32) have identified five basic 
approaches to program review: consultant-oriented, survey- 
oriented, data-oriented, self-study, and combination. The main 
characteristic of consultant-oriented reviews is evaluation is 
conducted by outside consultants. Success is dependent on the 
quality of the consultants and their ability to formulate 
practical re commend at ions. 

The primary characteristic of the survey-oriented review is 
the use of questionnaires to discern perceptions of the program 
as indicated by the program's faculty, students, employers of 
students, and alumni. The data-oriented approach emphasizes the 
collection and analysis of both quantitative and qualitative 
data. This approach is often used as a screening tool to 
identify programs needing more in-depth approaches, such as 
consultant-oriented or survey-oriented reviews 

Self-study reviews are generally formative in nature, 
focusing on program improvement. The major feature of this 
approach is that the program faculty are primarily responsible 
for conducting the review of their own program. Combination 
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reviews tend to "mix and match one or more of the other types to 
meet the specific needs of the institution conducting the review" 
(1990: 32) . 

Most approaches to program review can be classified as 
either formative or summative. In formative evaluation, 
assessments are made during program operation for the purpose of 
improvement . Summa t ive evaluation assesses the final worth of a 
program in terms of attainment of objectives for decisions 
pertaining to the program's future (Herman, Morris, and Fitz- 
Gibbon, 1987) • 

Clowes indicated that i;he preferred approach to community 
college program review should be formative, resulting in program 
improvement through the (1) identification of program needs, (2) 
appropriate application of resources, and (3) curricular and 
staff revisions. Summative evaluations designed to make life or 
death decisions about programs should not be used. Reviews using 
this approach tend to be counterproductive to the consensus style 
of management desired in the collegial setting of a college 
(Clowes, 1981:3-9). 

Finally, a comprehensive approach to evaluation is 
recommended by accrediting agencies. This approach assesses an 
educational institution's processes and resources as we 11 as the 
results of the education, including plans for the improvement of 
it's academic offerings ( Criteria for Accreditation . 1989-90:13). 
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Program Review Models 

A model is a simplified illustration of reality. It is 
simplified because it never depicts the full complexity of 
reality (Daft, 1983:21). Models are useful to program reviewers 
in that they provide a conceptual construct for designing 
evaluation protocols. By way of a model, the following protocol 
attributes are explicitly stated or implied: the philosophy, the 
evaluative criteria, and the interrelationships among its 
elements (Borchers , 1986:30) . 

Program review, as an area of study, has been researched 
extensively over the last twenty years. Curriculum evaluation 
models of the late 1960's and early 1970's wer3 developed to 
assist instructional designers in their efforts to conduct 
comparative summative evaluations. Such evaluations v/ere 
utilized by federal policy-makers to subctantiate accountability 
for the social reforms of that period. Th models stressed 
quantitative measures of evaluation, and were considered by many 
to be insensitive to the "unique characteristics and processes 
within local settings and issues as perceived by the 
stakeholders" (Herman, Morris, and Fi tz-Gibbon , 1987:9). 
Contemporary models have addressed the issue of being responsive 
to local variations in program implementation. These models 
emphasize qualitative methods to evaluate the effectiveness and 
efficiency of programs. 



Finley (1988:50-55) provides two general models used in 
evaluating two-year college programs: the quantitative measures 
model and the outcomes related model. The quantitative model, 
used to assess a program's cost effectiveness, is based on the 
systems analysis approach to evaluation previously mentioned. 
Example of evaluative criteria include: (1) program retention of 
students; (2) program FTE/headcount ; (3) attraction of new 
students; (4) training-related placement; (5) number of degrees 
awarded; and (5) average unit cost of the program. The 
outcomes related model, used to determine program effectiveness, 
is based on the behavioral objectives approach to evaluation 
previously mentioned. Finley concludes that a model should not 
be selected on the basis of ease of its implementation. Father, 
the model selected should be based on the needs and resources of 
the college. Additionally, colleges should not attempt to adopt 
a model in its entirety. Depending on the needs of the college, 
components from several compatible models should be selected; in 
other words, adapt, not adopt (Finley, 1988:50-56). 

The integrated feedback model is based on the survey- 
oriented approach to evaluation previously mentioned. Two 
variations of this model are using student input and using 
advisory committee input. Astin (1982:10-15), an advocate of 
using student input, implies that a student-oriented model of 
evaluation demonstrates that the college is aware of students' 
perceptions of the program's impact on their lives. Raulf and 
Ayers (1987) concur with Astin that the consumers of an 
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institution's prcgrams are students, and that their opinions 
should be carefully considered. Garrity (1984:40) outlines a 
model of using advisory committee input as a component of program 
review. In this model, advisory committees ensure the relevance 
of the program by reviewing program objectives, course 
requirements , and course content . 

Conrad and Wilson (1985:20-29) state that most insti- 
tutions employ variations of one of the following models: 
1 • In the goal -based model, the major elements of the 
evaluation protocol are the identified goals, objectives, and 
criteria used to appraise relative success or failure. 

2. The responsive model, the major components of the 
evaluation protocol focus more on program activities than on the 
program's stated goals and objectives. The model is organized 
around the interests of stakeholding audiences i.e., program 
information is collected, analyzed, and interpreted in light of 
the concerns of audiences that have a stake in the process. 

3. The decision-making model is used for the purposes of 
decision-making and accountability. The prototypical decision- 
making model is Stuf f 1 ebeam ' s Context-Input-Process-Product 
(CIPP) model. In this model, the variety of decisions that must 
be made require a variety of evaluation activities. Four types 
of evaluation were identified, each matching a category of 
decision: (1) context evaluation to assist decision-makers in 
establishing goals and objectives; (2) input evaluation to aid 
decision-makers 'in clarifying the methodology of achieving 
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program goals and objectives; (3) process evaluation to provide 

decision-makers with a feedback loop; and (4) product evaluation 

to provide decision-makers with information necessary to 

continue, modify, or terminate a program ( S tuf f 1 ebeam » 1971:20- 

26). The CIPP model is structured to meet managerial information 

needs via the provision of timely and relevant information. 

4. The connoissuership model is designed in harmony with 

the expectations of those served by the program review. 

The connoisseur alone guides the evaluation. Operating with 
few restraints, the evaluator is initially a processor of 
information, collecting data in whatever way he prefers. At 
the same time, however, he is constantly judging the program 
under review in much the same way that the art critic judges 
a work of art. The final report is based on the subjective 
judgement emanating from the connoisseur's own though 
processes, on his own construction of reality (Conrad and 
Wilson, 1985:28-29) . 

The Illinois Community College Board (1987) recognizes four 
basic models of community college program review: 

1, The screening model begins with an annual screening of 
statistical data available on all instructional programs. The 
results of this process are used to select programs requiring 
either a more focused review of programmatic problems or a 
comprehens i ve review. 

2. In the proportional mo del, colleges preselect 20 
percent of their programs for an annual comprehensive review. 
The unit under review conducts a detailed self-study. The self- 
study is then evaluated by either a college-wide committee or an 
external review team, which makes recommendations for action to 
the board of trustees. 
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3. The divisional model is similar to the porportional 
model in that a detailed self-study is conducted by the unit 
under review, which is then examined by individuals external to 
the program. However, rather than preselecting 20 percent of its 
programs for review, the college selects a particular 
organizational division within the institution to review each 
year . 

4. The combination model integrates both the screening and 
proportional models of evaluation. While 20 percent of the 
institution's programs are reviewed annually, some programs are 
reviewed comprehensively, others receive a focused review. 

Program review is a complicated undertaking. No single 
model will provide a complete evaluation. No ideal model of 
program review exits. Successful program reviews are useful to 
their intended audiences, practical to implement, and technically 
accurate (Herman, Morris, and Fitz-Gibbon, 1987; Resource Manual 
on Institutional Effectiveness , 1989). 

Common Elements In Successful Evaluations 

Several commonalities among successful program evaluations 
have been noted in the review of the related literature. Arns 
and Poland (1980) observed the following: 

1. No fixed protocol exists. Successful reviews recognize 
that no two programs are identical. 

2. The evaluation process includes both individuals within 
the program and persons accountable for the program. 
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3. Successful protocols include a self-study component, 
peer coordination, external review, and open, honest, and direct 
communication. 

One researcher found the following elements common in 
successful evaluat ions, regard less of the approach taken : ( 1 ) 
clear goals; (2) congruent activities; (3) satisfactory 
horizontal and vertical mobility for student transfer; (4) a 
thread of unity — cultural, social, or economic; (5) the effective 
use of personnel, including adjunct faculty; and (6) cost 
effectiveness (Clowes, 1981:3-9). Other researchers classified 
successful elements as aspects of the program to be evaluated, 
such as context characteristics, participant characteristics, 
processes in program implementation, program outcomes, and 
program costs (Herman, Morris, and Fitz-Gibbon, 1987:22). 

Larson (1985:106) listed operating characteristics of 
successful program review protocols. Successful protocols: (1) 
assess the quality of program results; (2) compare program 
objectives with program results; (3) provide information for 
revision of program goals and/or content, planning and budgeting, 
and the reallocation of resources to oversubscribed programs; and 
(4) are formally approved by the faculty. 

According to Barak and Breier (1990:5), the following 
factors have been shown to be crucial in conducting a successful 
review: fairness, comprehensiveness, timeliness, good 
communication, objectivity, credibility, and utility. Evaluation 
systems must be perceived as being fair. To facilitate 
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fairness, a program review protocol should include particular 
f eatures : 

It should have an explicated set of procedures. These 
procedures should be publicly known. These procedures and 
rules should be consistently appl ied . There should be 
provisions for confidentiality of information, and these 
should be understood by all from the beginning. There 
should be restrictions on the use of informal i nf or mat ion, 
that is, considerations not included in the evaluation 
reports. There should be avenues for recourse and redress. 
S trong admi ni strative support is necessary if these 
procedures are to be carried out equally with all parties 
(House, 1982) . 

Accrediting bodies have published several principles found 
in successful program review efforts. It is essential to involve 
those individuals who will be affected by and/or responsible for 
the evaluation. It is important to determine the appropriate 
level of analysis. Program review protocols must relate to the 
level or unit responsible for implementing recommendations. 
Another principle concerns e inclusion of comparative data. In 
the absence of absolute standards for judging the worth of a 
program, the use of comparative data becomes critical. Finally, 
colleges must take proactive measures to assist the faculty and 
other constituents to become more fully committed to the program 
review process. This is accomplished when (1) college 
administrators expect departments to support program review 
recommendations and (2) resources are allocated on the basis of 
strategic plans and budgets justified by the eva-uation ( Resource 
Manual on Institutional Effectiveness . 1989: 17) . 

In summary, program review protocols should be custom- 
designed to the specifications of the particular college. 

ERLC ^ ' 
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Program participants , including faculty, students, and 
administrators, should participate in the process. Table One 
provides a listing of the common elements in successful 
evaluat ions . 

Table One 

Common Elements in Successful 
Evaluat i ons 



Fai rness 
Timeliness 
Clear goals 
Ob j ect ivi ty 
Credibi 1 ity 
Comprehens i venes s 
Cost effectiveness 
Congruent activities 
Utilization of results 
Effective use of staff 
Use of comparat i ve data 
Approved by the faculty 
Provides information for decision-making 
Relates to the unit responsible for implementation 



Criteria Used in Program Review 
"Criteria are the dimensions in terms of which the program 
is to be judged" (Petrie, 1982:22) • Most colleges involved in 
academic program review use evaluative criteria to some degree 
(Barak, 1982). Institutions can employ one or more of the 
following techniques to define criteria used in program review: 
small group discussion, large group discussion, delphi surveys, 
copying from others, outside consultant, and administrative 
decision (Barak and Breier, 1990). While no set rules for 



13 



18 



establishing criteria used in program review exist, several 
issues must be considered, such as the kind of criteria to be 
used, the type of criteria to be used, and criteria relating to 
institutional effectiveness (Wallhuas, 1982; Caruso, 1985; 
Resource Manual on Institutional Effectiveness . 1989). 

Four major kinds of criteria are cited in the professional 
literature: mi s s i on/cent ra 1 i ty , quality, cost, and demand 
(Caruso, 1985; Mortimer, 1982; Dougherty, 1980); Barak and 
Engdahl, 1978). Table Two provides examples for each kind of 
criteria. 



Table Two 



Examples of Kinds of Criteria 



Kind 



Examples 



mission/ cent rality 



The program is consistent with college 
goals and mission. 



qual i ty 



Of faculty; students; curriculum; 
facilities; equipment; library holdings; 
administration. 



cost 



Cos t /revenue relationship; benef i t s to 
the students, the college, and society; 
faculty; facilities; equipment ; 
enro 1 Iment . 



demand 



Past, present, and projected future 
enrollment; demand for graduates; job 
opportunities for gradua t es ; s tudent 
interest; justification of need; 
comparat i ve advantage to other similar 
programs offered in the service area; 
benef it to society. 
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Two major types of criteria cited in the professional 
literature are quantitative and qualitative (Barak and Breier, 
1990; Caruso, 1985; Mingle, 1981; and Dougherty, 1980). 
Quantitative criteria are descriptive in nature and are concerned 
with program effects such as degree productivity, student credit- 
hours (numbers and cost per), and faculty workloads. Qualitative 
criteria are less specific, assessing the quality of students, 
faculty, administration, curriculum, facilities, and equipment. 
Regardless of the type and kind of used, the criteria should be 
developed to meet the college's needs and the data available to 
program revi ewers. 

The Southern Association of Colleges and Schools (SACS) 
provides a list of student outcomes frequently employed as 
evaluative criteria in the assessment of institutional 
effect i venes s : 

retention and completion rates; student achievement in 
general education; student achievement in their major field; 
student perceptions of their development toward educational 
objectives; student affective development; opinions of 
program quality given by students, alumni, employers, and 
drop-outs; job placement rates; performance after transfer 
from two-year to four-year institutions; external 
recognition of achievements of students and graduates 
( Resource Manual on Institutional Effecti veness. 1989: 10) . 

Other indicators of institutional effectiveness include: 

an adequate enrollment to support the organization; sound 
fiscal health; a positive public image; an attraction for, 
and ability to hold, a strong management team; prudent use 
of human and material resources; ef f ect iye coordinat ion of 
various programs and activities with a minimum of 
duplication, with strong mutual support, and a minimum of 
nonproductive internal conflict; application of the 
institution's management talent to high-priority 
opportunities and problems; and assists administrators in 
applying their highest skills to opportunities in their 
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areas of responsibility for the largest portion of their 
time (Haag^Mut ter , Holzapfel, and Jones, 1988:101). 

Barak and Breier indicate that indivi dua 1 s conduc ting their 

first program review usual y collect too much information. Such 

rev i ewers f ind themselves " incapable of effectively analyzing the 

information because of its quantity" (1990:27). In general, 

three to six good criteria supported by data are sufficient for 

most appl icat ions . 

Processes Used in Program Review 

Three major issues emerge from the writings of others 
concerning the processes used in program review. The first issue 
relates to determinations that must be made in designing the 
program review protocol. According to Wallhaus, these decisions 
characterize the evaluation process and determine its acceptance 
and effectiveness. During this design phase, institutions must 
determine: (1) the purposes and objectives of evaluation; (2) the 
scope and focus of evaluation; (3) the scheduling and timing of 
evaluations; (4) the criteria used in evaluation; (5) the roles 
and responsibilities of participants in the process; and (6) the 
use of evaluation results (Wallhaus, 1982:75). The second issue 
emerging from the writings of others concerning program review 
processes relates to the actual steps in designing the 
evaluation. Seitz provided the following steps to facilitate a 
program review process: 



1. Define the scope. 
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2. Specify inputs, activities, and outputs. 

3. Determine the types of data to be compiled. 

4. Identify the support services and personnel to be 
invo 1 ved in the evaluation. 

5. Schedule, carefully and consistently. 

6. Co' lect and compile data. 

7. Compare, critique, and analyze. 

8. Draw conclusions (Seitz, 1981:57), 

Barak and Breier identified two phases in designing a program 
review protocol. The first phase involves the conducting of a 
needs assessment which is used to determine the purposes, 
objectives, and needs with respect to program review. The needs 
assessment can be conducted by committee, an administrative team, 
or extrainstitut ional consultants (1990:13-15). 

The second phase involves the conceptualization of the 
protocol which determines who will be involved in the process, 
what their role will be, when the review will occur, and how the 
review is to be conducted. During this phase, the criteria for 
evaluation are developed, as well as the format for the final 
report (1990:20). The typical final report consists of the 
foil owing sections: 

1. The description of the reviewed program; 

2. An explanation of the process used, i.e., the who, 
what, where, when, and how of the review process; 

3. The results of the review process, including all 
support i ng document at ion; and 

4. Recommendation for program improvement and a schedule 
for implementation of recommendations (1990:55). 
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Caruso stated that the design phase of a program review 
process is important to its credibility, legitimacy, and success. 
He developed the following steps of a program review process: (1) 
define the purpose; (2) develop the methodology; (3) develop the 
criteria to be used; (4) develop guidelines and procedures for 
conducting the review; (5) conduct the review; (6) formulate 
recommendation based on the findings of the review; and (7) 
implement the recommendations (Caruso, 1 985 : 1 85- 1 86 ) ♦ 

The final issue emerging from the writings of others 
concerning program review processes relates to the roles and 
responsibilities of participants. The delegation of duties and 
responsibilities Is necessary to avoid confusion and chaos in the 
process. A survey of college presidents indicated that the 
process used in their institutions was perceived as being both a 
formative and a summative process (Borchers, 1986), 

Barak and Breier (1990) have identified typical 
responsibilities of faculty and administrators involved in a 
program review process. If a formative approach to program 
evaluation is used, the faculty design the review; conduct the 
self study; collect the data; survey students, employers, and 
alumni ; ana lyze the data; write the report; and impl ement 
recommendations. Administrators are simply interviewed. If a 
summative approach is used, faculty in the program under review 
recommend consultants; conduct the self-study; and assist in the 
collection of data. Faculty outside of the program oerve on the 
review committee; collect and analyze the data; survey student, 
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employers of students, an6 alumni; write the final report and 
make recommendations. Line administrators hire the consultants; 
appoint the review committee; design the protocol; review the 
recommendations and develop a plans for implementation. Staff 
administrators assist in collecting and analyzing the data (Barak 
and Breier, 1990:36) . 

Utilization of Results 

The major issue concerning the utilization of trie results of 

program review is that institutions have an obligation to their 

constituents to use such results in a systematic planning and 

evaluation process r Criteria for Accre ditation. 1989-90). House 

(1982) urges decision-makers to ensure that something happens as 

a result of program review. Perhaps the most devastating outcome 

of a program review process is inaction. The intended effects of 

participation in a program review process are program change, 

innovation, or improvement; otherwise, the process has not been 

used (Barak and Breier, 1990). SACS states: 

In the absence of commitment to use evaluation results, all 
previous steps in the planning and evaluation process would 
become little more than futile exercises which institutions 
can ill afford, and the institution's evaluation process 
could not be considered adequate (Resource Manual on 
Institutional Effectiveness . 1989:11). 

The most common reason for failure in a program review 
process is "the well-founded perception of the faculty tjj^t the 
whole effort has been a waste of time, since nothing has happened 
as a result" (House, 1982:53). Borchers (1986) provides several 
motives of low use of program review results: lack of 
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dissemination of the report; insufficient financial resources and 
staff dedicated to the process; fear of economic loss by- 
participants; job security; and poor design of the program review 
process^ Conrad and Wilson concur, offering four general reasons 
for the non-use of program review results; organizational 
inertia; inadequate process design; lack of consensus regarding 
the need for some reviews; and the "multiple sources of 
information competing for the attention of decision-makers" 
(Conrad and Wilson, 1985:57). 

Using the results of a program review process brings a sense 
of closure to participants. As previously mentioned, 
recommendations are made, namely, to continue, modify, or 
discontinue the program. These recommendations are based on the 
assessment of program quality. However, another benefit accrues 
to institutions that utilize the results of evaluation: enhanced 
communication. Larson (1985) proposed that program faculty were 
educated by this enhanced communi c?^t i on , implying that had not a 
review process occurred, the need for change would not be 
acknowledged. He offers the following examples to describe this 
effect : 

Making faculty aware of the difficulty of fair evaluation; 
better communication about realities of program costs and 
outcomes; faculty awareness or* the need to "look at what we 
do at the college"; consideration by faculty of "what they 
are doing and where they are going"; a forced inspection of 
instructional programs by faculty (Larson, 1985:109). 

Finally, most researchers conclude that program review is 

beneficial. However, since only a few studies have 
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systematically analyzed the effects of program review, the basis 
for this conclusion is weak. Additionally, some studies indicate 
that program reviews do not achieve their desired results (Barak 
and Breier, 1990) . 

Add i t i ona 1 Comment s 

Several minor issues relating to program review were cited 
in the professional literature. The more prominent were: common 
myths, political realities, the state role in the process, and 
the link to accreditation. 
Common Myths 

Barak (1986) indicated several common myths which developed 
during the past decade. The first myth is that academic program 
review is not necessary if the program is either regionally 
accredited or accredited by a recognized specialized agency. 
Attainment of accreditation is considered only one outcome of a 
successful program review process. Another myth is that there is 
only one correct method to conduct an evaluation study. In 
reality, there are as many ways to conduct program reviews as 
there are program reviewers. Another myth is that program review 
will produce spectacular results. Most program review processes 
result in minor change that take several years to implement. 
Po 1 i 1 1 cal Real i t i es 

Stevenson ( 1985) investigated a self--study program review 
process of a public community college. One aspect of the study 
focused on the political realities of the process as perceived by 
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faculty and instructional administrative stakeholders. The 
following is a summary of conclusions concerning faculty 



s takeho Ider s : 

Conclusions derived from the data indicate that the majority 
of faculty stakeholders view the instructional review 
process as a part of their professional responsibility. 
Consequently, most elected to actively participate in the 
process. The underlying motivation was, however, to monitor 
and to control the process. All faculty stakeholders viewed 
the process as a threat to their professional expertise and 
territory. Most faculty efforts to guide the process and 
outcomes were proactive. There was, however, a small group 
of faculty stakeholders who were dissident and tried to 
subvert the process. All efforts by faculty stakeholders, 
both proactive and dissident, were directed toward 
influencing decision outcomes that either maintained the 
status quo or reflected their emerging interests (Stevenson, 
1985: 166) . 

The following is a summary of conclusions concerning admin- 



istrative stakeholders: 

Division administrators, academic middle managers, 
effected wide-spread decision outcomes. They had access to 
multiple resources in order to bring about change. In 
general, the desired outcomes were achieved through 
influencing the faculty stakeholders. In those instances 
when the desired outcomes were not forth- coming, coalitions 
formed with upper level management or selected faculty 
stakeholders. Decision outcomes that emerged from the 
review process inlluenced by the division administrator were 
moderate to extensive. 

When significant program changes were necessary, the 
division administrators provided the impetus and the 
oversight for the changes. Faculty stakeholders, on the 
other hand, attempted to maintain the status quo throughout 
the process (Stevenson, 1985:167). 

Stakeholders who understood the political structure of the 
college used that knowledge for their benefit. 
State Eole in Assessment 

A report published by the Education Commission of the States 
(1986) recognizes the role of the state in program evaluation as 
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the mover and shaker, with the role of the college being process 

design and implementation. In other words, the s' te is 

responsible to stimulate program review action and ensure that 

the process is carried out. A key issue here is for 

institutional and state-level interests to converge so that a 

single review process can achieve the goals of both. Such 

cooperative endeavors are called shared reviews (Floyd, 1983). 

Link to Accreditation 

Accreditation, granted to a college that meets or exceeds 

stated criteria for educational quality, has two basic purposes: 

to assure quality and to assist in improvement. The accrediting 

body encourages improvement by requiring institutions to conduct 

periodic self-studies to identify strengths, weaknesses, and 

develop strategic plans to ameliorate the identified weaknesses 

(Ellison and Smith, 1987). Program review is a component of such 

a strategic plan. According to SACS (1989-90:9): 

The effectiveness of self-regulatory accreditation depends 
upon an institution's acceptance of certain 

responsibilities, including institutional involvement in and 
commitment to the accreditation process. This process 
assumes that each member institution has the responsibility 
to participate in and to accept an honest and forthright 
assessment of institutional strengths and weaknesses. 

As previously stated, accrediting associations emphasize that 

quality is assessed by the degree of conformance between an 

institution's goals and actions. The basis for this statement 

lies in the assumption that many colleges with diverse programs 

exist, "and not all of them ought to behave in exactly the same 

way, given their varied purposes" (George, 1982:47). 
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In conclusion, the survey of literature relating to academic 
program evaluation processes contained seven theme n : purposes of 
program review; different approaches to program review; program 
review models; common elements in successful evaluations; 
criteria used in program review; processes used in program 
review; and utilization of results. Although program review is 
not considered to be an exact science, these seven themes provide 
insight into contemporary program review processes. 
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